The Use of Behavior Models for Predicting Complex Operations 
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ABSTRACT: Modeling and simulation (M&S) plays an important role when complex human-system notions are 
being proposed, developed and tested within the system design process. National Aeronautics and Space 
Administration (NASA) as an agency uses many different types of M&S approaches for predicting human-system 
interactions, especially when it is early in the development phase of a conceptual design. NASA Ames Research 
Center possesses a number of M&S capabilities ranging from airflow, flight path models, aircraft models, scheduling 
models, human performance models (HPMs), and bioinformatics models, among a host of other kinds of M&S 
capabilities that are used for predicting whether the proposed designs will benefit the specific mission criteria. The 
Man-Machine Integration Design and Analysis System (MIDAS) is a NASA ARC HPM software tool that integrates 
many models of human behavior with environment models, equipment models, and procedural / task models. The 
challenge to model comprehensibility is heightened as the number of models that are integrated and the requisite 
fidelity of the procedural sets are increased. Model transparency is needed for some of the more complex HPMs to 
maintain comprehensibility of the integrated model performance. This will be exemplified in a recent MIDAS v5 
application model and plans for future model refinements will be presented. 


1. Introduction 

Complex system integration issues require that the 
model development process generally follow an 
iterative design philosophy that collaboratively 
leverages empirical human data (i.e., either human in 
the loop, HITL, simulations or real-time 
measurements) and concurrently feeds information to 
HITL simulation processes. Many organizations are 
faced with the goals of completing research as 
efficiently as possible while maintaining acceptable 
levels of safety to successfully complete a mission. 
NASA is no exception. Modeling and simulation 
techniques, particularly human behavior models, play 
an important role when complex human-system notions 
are being proposed, developed, and tested across many 
of the ten NASA centers. For instance, NASA 
Johnston Space Center (JSC) utilizes M&S to represent 
environments, physical structures and equipment 
components, crew stations, planets and planetary 
motions, gravitational effects, illumination, human 
anthropometric and biomechanics, among a host of 
other domains. NASA Ames Research Center also 
possesses a number of M&S capabilities ranging from 
airflow, flight path models (e.g., Airspace Concept 
Evaluation System, - ACES), aircraft models, 
scheduling models (e.g., Core-XPRT, Science Planning 
InterFace to engineering - SPIFe), human performance 
models (HPMs), and bioinformatics models, among 
many other kinds of M&S capabilities. One of the 
many NASA M&S capabilities, an ARC-related HPM 


capability termed the Man-Machine Integration Design 
and Analysis System (MIDAS) is highlighted because 
of its relevance to the field of human behavior 
representation. 

1.1 Human Performance Models (HPMs), Concept 
Development and Testing 

Modeling can play a role in all phases of the concept 
development, refinement, and deployment process. 
Hybrids of continuous-control, discrete-control and 
critical decision-making models represent the 
'internal models and cognitive function' of the 
human operator in complex control systems, and 
involve a coupling among humans and machines in 
a shifting and context sensitive environment. These 
models, known as HPMs, have arisen as viable 
research options due to decreases in computer costs, 
increases in representative results, and increases in 
model validity. They are especially valuable because 
the computational predictions can be generated early in 
the design phase of a product, system or technology to 
formulate procedures, training requirements, and to 
identify system vulnerabilities and where potential 
human-system errors are likely to arise. The model 
development process allows the designer to formally 
examine many aspects of human-system performance 
with new technologies to explore potential risks 
brought to system performance by the human operator 
(Gore & Smith, 2006). Often this can be accomplished 
before the notional technology exists for human-in-the- 



loop (HITL) testing (Gore, 2000). This method 
possesses cost and efficiency advantages over waiting 
for the concept to be fully designed and used in 
practice (characteristic of HITL tests). Using HPMs in 
this manner is advantageous because risks to the 
human operator and costs associated with system 
experimentation are greatly reduced: no experimenters, 
no subjects and no testing time (Elkind et al., 1989; 
Gore, 2000). Hooey and Foyle (2008) outline that 
HPMs can be used to conduct system robustness 
testing to evaluate the system from the standpoint of 
potential deviations from nominal procedures to 
determine the impact on the performance of the human 
and the system (“what-if ’ testing). 

1.2 The Man-machine Integration Design and 
Analysis Systems (MIDAS) 

MIDAS is a dynamic, integrated human performance 
modeling and simulation environment that facilitates 
the design, visualization, and computational evaluation 
of complex man-machine system concepts in simulated 
operational environments (Gore, 2008). MIDAS 
combines graphical equipment prototyping, dynamic 
simulation, and HPMs to reduce design cycle time, 
support quantitative predictions of human-system 
effectiveness, and improve the design of crew stations 
and their associated operating procedures. HPMs like 
MIDAS provide a flexible and economical way to 
manipulate aspects of the operator, automation, and 
task environment for simulation analyses (Gore, 2008; 
Gore, Hooey, Foyle, & Scott-Nash, 2008; Hooey & 
Foyle, 2008). 

Gore & Smith (2006) outline that MIDAS links a 
virtual human, comprised of a physical anthropometric 
character, to a computational cognitive structure that 
represents human capabilities and limitations. The 
cognitive component is comprised of a perceptual 
mechanism (visual and auditory), memory (short term, 
long term-working, and long term), a decision maker 
and a response selection architectural component. The 
complex interplay among bottom-up and top-down 
processes enables the emergence of unforeseen, and 
non-programmed behaviors (Gore & Smith, 2006). 
MIDAS can suggest the nature of pilot errors, and 
highlight precursor conditions to error such as high 
levels of memory demand, mounting time pressure and 
workload, attentional tunneling or distraction, and 
deteriorating situation awareness (SA). 



Figure 1. MIDAS’ Environment, Task, and 
Anthropometric Models. 

MIDAS can be used as a cognitive modeling tool that 
allows the user to obtain both predictions and 
quantitative output measures of human performance, 
such as workload and SA and as a tool for analyzing 
the effectiveness of crew station designs, information 
display concepts, operator roles and responsibilities 
from a human factors perspective (Gore, 2008). 
MIDAS has proven useful for identifying general 
human-system vulnerabilities and cross-domain error 
classes and for recommending mitigation strategies and 
job re-designs to account for the vulnerable areas, or 
risks, in system design (Gore & Smith, 2006). 
Fundamental design issues can therefore be identified 
early in the design lifecycle, prior to the use of 
hardware simulators and HITL experiments. In both 
cases, MIDAS provides an easy to use and cost 
effective means to conduct experiments that explore 
"what-if questions about domains of interest. 

1.3 The MIDAS User Interface Assists 
Comprehensibility 

MIDAS v5 has a graphical user interface 1 (GUI) that 
does not require advanced programming skills to use. 
The GUI brings many of the previously embedded 
functions to the surface so that the model analyst can 
observe the underlying structure as well as the model’s 
operation as it is run. The integrated GUI enables the 
user to build human procedures from MIDAS primitive 
tasks, create their own tasks, incorporate a series of 
nested procedures, change the SA context during the 
simulation and manipulate visual and auditory 


1 MIDAS uses Microsaint Sharp as its GUI which uses 
the C-Sharp programming language 



attributes of equipment components. The MIDAS 
analyst can organize the human-system interactions 
visually, thereby greatly improving the model’s 
transparency. Other features of MIDAS v5 include 
dynamic visual representations of the simulation 
environment, support for multiple and interacting 
human operators, distributed simulation, monte- 
carlo/stochastic performance, HPM timelines, task 
lists, workload, and SA, performance influencing 
factors (such as error predictive performance, fatigue 
and gravitational effects on performance), libraries of 
basic human operator procedures (how-to knowledge) 
and geometries for building scenarios graphically (that 
leverage heavily from Siemens' Jack software). 

1.4 MIDAS Approach and Land Applications 

The current air traffic control (ATC) system will not be 
able to manage the predicted two to three times growth 
in air traffic (JPDO, 2009). The Next Generation Air 
Transportation System (NextGen) is a future aviation 
concept that has as its goals to significantly increase 
the capacity, safety, efficiency, and security of air 
transportation operations (JPDO, 2009). 

MIDAS v5 has been applied to examine a NextGen 
approach to land concept termed the very closely 
spaced parallel approach (VCSPA). In order to 
evaluate this concept, two MIDAS v5 models were 
generated. The first was a current day Simultaneous 
Offset Instrument Approach (SOIA) model that 
contained the current day procedures and the second 
was a NextGen VCSPA model that contained 
predictive displays in the cockpit and a modification to 
the roles and responsibilities of the flight crew and 
ATC modeled operators. This simulation involved over 
500 tasks and culminated in a verifiable model of 
approach and land operations (vetted by Subject Matter 
Experts - SMEs). The SA model was augmented 
within MIDAS to represent how a cockpit crew builds 
SA of traffic, terrain, and weather information given 
the accessibility of sources of information. This model 
effort illustrated the “what-if’ simulation capability 
within MIDAS. The “what-if’ approach was completed 
when MIDAS was exercised with one set of displays 
and procedure sets designed to represent current day 
operations and roles followed by a second simulation 
with an alternate set of displays and procedures 
encoded to represent the NextGen displays and 
expected procedures. The model underwent an iterative 
verification/validation process that included 
examining: (1) the task sequences and the performance 
of the model as it executed; (2) the visual fixations, 
task timings, and workload relative to expected 
performance given the inputs to the model; and pilot 
performance according to SME evaluations. 

Model comprehensibility is defined as understanding 


the relationships that exist among the models being 
used in an application, the performance of the models 
in the application, which models are being triggered in 
the model architecture, and whether the model is 
behaving as the model analyst would expect. MIDAS 
v5’s comprehensibility was greatly improved with the 
transparent model architecture (Gore, 2008). The 
operation of this complex model was verified 
throughout development and was validated according 
to SME evaluations. The verification phase of the 
model was improved given the visibility into the 
model’s operations at any given point in simulation 
time combined with the cross checking of the jack 
visualization and the simulation runtime data that was 
output. The comprehensibility of this model would not 
have been possible without such a transparent 
architecture. 

This MIDAS v5 effort lead to a greater awareness of 
potential parameters that should be included in system 
designs and enabled the research program to visualize 
the interactions that will be likely in future NextGen 
operations. It is anticipated that a formal validation 
approach will be developed and applied to the VCSPA 
model in an upcoming Federal Aviation Authority 
(FA A) task. This FAA task will require model 
refinement and validation, an increased number of 
alternative closely spaced operations for additional 
what-if scenarios including alternative pilot roles and 
responsibilities, and information requirements. 

2. Conclusion 

A number of significant challenges exist for the state of 
the art in FTPMs, two of which will now be highlighted. 

Transparency. The first challenge relates to model 
transparency. Model transparency refers to the ability 
to comprehend the relationships that exist among the 
models being used in the simulation, the performance 
of the models in the simulation, which models are 
triggering in the model architecture, and whether the 
model is behaving as the model developer would 
expect (Gore, 2008). Other researchers refer to this as 
model traceability, model behavior visibility, model 
verifiability, and model interpretability (Elkind et al., 
1989; Napiersky, Young, Harper, 2004; Gluck & Pew, 
2005; Hooey & Foyle, 2008). Transparency in 
integrated HPMs is needed to support model 
verification, validation, and credibility. However, 
model transparency can be difficult to attain because of 
the complex interactions that can exist among the 
cognitive, physical, environment and crew station 
models, and because the cognitive models embedded 
within integrated HPMs produce behaviors that are not 
directly observable. This paper illustrates how the 
transparency-related augmentations to MIDAS v5 have 
improved model comprehensibility. 



The type of transparency that is needed to accurately 
interpret model output remains undefined. Increased 
transparency is always possible, but there comes a 
point when transparency reduces model 
comprehensibility. 

Validation. The second challenge facing the HPM 
community is validation. Validation remains a very 
large challenge for the HPMs community because 
statistical validation is oftentimes seen as the Holy 
Grail for determining whether a model is suitable but 
when models are deemed statistically valid, they are 
less generalizable, and less re-usable for applications in 
new contexts. This places the field of modeling into the 
conundrum of making models that are statistically valid 
(correlation, r=.99) but that lack the ability to 
generalize to other tasks or scenarios. When the 
generalizability of the model is limited, then its value 
as a cost-effective approach to predict complex human- 
system interactions is reduced. 

Validation is further challenged when modeling future 
technology concepts where no or little HITL data exists 
upon which to statistically validate a model (as in the 
NextGen aviation systems or concepts being designed 
for the Space program). It is argued that our definition 
of model validation must be expanded beyond that of 
statistical results validation to be more representative 
of a model develop-model verify-model manipulate - 
model validate iterative process. 
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